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2 Algorithms, Conventions and 
New Regulation Processes 


Rainer Diaz-Bone and Simon Schr6ér 


Introduction: Algorithmic Governance 


Contemporary societies are data worlds. Digital devices, software and 
the Internet transform individuals’ lifeworlds, personal communica- 
tion, the economy (i.e., production processes, distribution and con- 
sumption), banking and credit, health care and health care insurance, 
public and private transport, law and the judicial system, policing and 
surveillance, social media and mass media and other private as well 
as public spheres into computerized realities and digital data. This 
transformation is grasped with notions (and buzz words) such as “da- 
tafication”, “data revolution” or “big data” (Mayer-Schénberger and 
Cukier, 2013; Kitchin, 2014; Mejias and Couldry, 2019; Peeters and 
Schuilenburg, 2021). Although many user interfaces and representa- 
tions on displays of digital devices (as smart phones, wearables, 
notebooks) are organized and offered in a non-numeric visual form 
(as images), the underlying data form is numerical. Nowadays, huge 
amounts of numerical data are generated, stored and analyzed mainly 
by big Internet companies, who detect behavioral patterns and exploit 
these to gain profits (Mayer-Schénberger and Cukier, 2013). In many 
countries, national security agencies gather data to track citizens’ ac- 
tivities or to detect crime (The Economist, 2016; Botsman, 2017). 

The notion of “big data” was originally invented to label amounts of 
data, which can no longer be stored on single computers (“volume”), 
which are produced and analyzed continuously (“velocity”) and which 
vary in data formats (“variety”; Lazer and Radford, 2017). The no- 
tion of big data has become a present marker in public debates for 
utopian perspectives of data-driven innovations and economic pro- 
gress (Mayer-Schénberger and Cukier, 2013; Mayer-Schonberger and 
Ramge, 2018) as well as for dystopian perspectives of surveillance and 
the control of individual behavior (Zuboff, 2019). 


DOI: 10.4324/9781003173427-2 
This chapter has been made available under a CC BY NC ND 4.0 license. 


Algorithms, Conventions, New Regulation 25 


Related to notions as big data is the term algorithm, which has long 
been established in computer sciences. In fact, algorithms are “ordi- 
nary” parts of contemporary ways of living; they are ubiquitous in 
digitized societies. “Digital technology is enabled as much by its hard- 
ware, the physical components that make up computers and digital 
devices, as by its software, the programs that run on it. The backbone 
of programs are the algorithms that they implement” (Louridas, 2020: 
xiv). 

However, algorithms have become an issue in the social sciences, but 
also as a public discourse element too (Steiner, 2012; Pasquale, 2015; 
Beer, 2016, 2017; O’Neil, 2016; Eubanks, 2017; Burrell and Fourcade, 
2021). The reason is that the power of big data unfolds only when it can 
be accessed by computer networks, which apply algorithms to detect 
patterns, and to automatically generate evaluations, predictions as 
well as decisions. More and more algorithms are applied to make im- 
portant decisions, which affect everybody’s everyday life. And many 
of these decisions can have negative and illegitimate effects, promote 
different forms of social inequality and result in unfair life chances 
(Fourcade and Healy, 2013; Eubanks, 2017; O’Neil, 2016). And it is this 
combination of big data and algorithms which enables its outreach of 
power effects to more and more domains and the emergence of new 
governance effects and governance forms: 


We are living in the midst of a significant transformation of our 
lives, and while it is an incredible time and place to be in, we must 
be wary of the effects that come along with it. Mind-boggling 
amounts of data are generated regarding our daily actions with 
algorithms processing and acting upon these data to make deci- 
sions that manage, control, and nudge our behavior in everyday 
life. The use of algorithms not only expands the possibilities of 
current control and surveillance, but also introduces a new par- 
adigm characterized by an increased rationality of governance, a 
shift in the functioning of power, and closure of decision-making 
procedures. We can refer to this by using the term ‘algorithmic 
governance’ — the replacement of human, legible and accountable 
judgements with ‘black-box’ algorithms [...]. Algorithmic govern- 
ance is central to the functioning of public and private organiza- 
tions. For instance, police forces use them to predict where, when 
and by whom crimes are more likely to be committed [...]. In crim- 
inal justice, algorithms are used to predict future dangerousness 
of defendants and convicts [...]. Marketeers use algorithms to ana- 
lyze consumer audiences from online search queries, credit card 
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purchase data, and behavioral data [...]. Government agencies are 
turning towards algorithms to, among other things, identify wel- 
fare fraud, deliver public services, allocate regulatory oversight 
resources, and assess risks in child protection [...]. Taken together, 
algorithms, machine learning and artificial intelligence form the 
new digital infrastructure of our society. 

(Peeters and Schuilenburg, 2021: 1/3/4)? 


It is evident that algorithms are based on norms and normative deci- 
sions, on programmers’ values and principles, how to quantify and 
how to categorize events, persons or objects (Desrosiéres, 2008; O’Neil, 
2016). As Peeters and Schuilenburg (2021: 4) emphasize, “algorithms 
can only exist in a stable environment of standardized codes and clas- 
sifications”, categories and quantifications are their “input” and (in 
many applications) also their “output”. Algorithmic governance there- 
fore can be conceived of as the power to classify and quantify persons, 
objects and events on the basis of social conventions how to categorize 
and to quantify. These classifications and quantifications are articu- 
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lated by different words such as “scoring”, “sorting”, “ranking”, “rat- 
ing”, “status determination”, “clustering”, “risk assessments” etc., and 
they all imply an evaluative and valorizing effect. These classifications 
and quantifications are in fact measurements, which are themselves 
built up on norms, values and — more broadly conceived — on societal 
rationalities and institutional logics. And algorithms themselves en- 
tail normative decisions and value-based criteria (as criteria, how to 
optimize or when to end the calculation).* This is not an argument to 
discredit or to dismiss measurements and algorithms, which both are 
inevitable in modern societies. But the argument points to the need to 
scrutinize the link between measurements, algorithms and values as 
well as its effects. 

For sociology, the analysis of power effects, released and advanced 
by datafication, measurement and algorithms is of core importance — 
as notions like “social power of algorithms” (Beer, 2017) or “algorith- 
mic power” (Lash, 2007; Peeters and Schuilenburg, 2021) highlight. 
But different additional positions and perspectives should be included 
in the analysis of the relation between datafication, measurement, al- 
gorithms on one side and society, power effects and governance on the 
other side. (1) It isimportant not to restrict the analysis to the power ef- 
fects only, but to open the black box of measurement and algorithmic 
calculation itself. (2) Also, sociological analysis has to recognize the 
plurality of data worlds, of normative orders and value systems, which 
influence datafication, measurement and algorithmic governance. (3) 
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Therefore, it is important not to assume coherent power effects and 
to include non-intended and countervailing power effects, resistance 
and social critique too. (4) Finally, if processes of datafication and al- 
gorithms are to be evaluated from a sociological point of view, there 
is a need to relate them to collective action and to common goods and 
to ask how datafication and algorithms enhance capacities and enable 
agency to approach social problems, to improve living conditions as 
well as social participation and in general to advance societies. 

In this contribution, the black box of these measurements is con- 
ceptually approached by relying on the institutionalist approach of 
economics and sociology of conventions (EC/SC). Also, this approach 
offers a pluralist perspective on data worlds, in which algorithms are 
differently developed, evaluated and applied. 

In the next section (“Economics of Convention — Quantification, 
Algorithms and the Common Good”), the approach of EC/SC is in- 
troduced. Then the notion of data worlds is introduced (“The Plurality 
of Data Worlds and Data Regulation”). These worlds allow a differen- 
tiation that enables an analysis framework for algorithmic regulations 
to be developed (“Analyzing Algorithmic Regulations: Critique in Sit- 
uations of Uncertainty”). Finally, the contribution applies the conven- 
tionalist framework to different examples of algorithmic norm setting 
and enforcement (“New Regulators — New Perspectives on Regulatory 
Processes”). 


Economics of Convention — Quantification, Algorithms 
and the Common Good 


EC/SC was originally developed in France in the analysis of socio- 
economic categories and official statistics but spread out to become 
an interdisciplinary and international neopragmatist institutional 
approach (Storper and Salais, 1997; Boltanski and Thévenot, 2006; 
Eymard-Duvernay, 2006a, 2006b; Diaz-Bone and Salais, 2011; Diaz- 
Bone, 2018). The most widely known model of conventions was 
worked out by Luc Boltanski and Laurent Thévenot (2006), who iden- 
tified the industrial convention, the market convention, the domestic 
convention, the convention of renown and the convention of inspira- 
tion. These conventions are deeper logics of quality assessment, but 
also for the critique and justification of worth. One neopragmatist 
core element is EC/SC’s awareness of the link between facts and val- 
ues, which is the link between norms and measurements (Desrosiéres, 
2008, 2014; Diaz-Bone, 2016, 2017, 2019; Diaz-Bone and Didier, 2016; 
Diaz-Bone and Horvath, 2021). Measurements are therefore not 
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impartial representations of a foregoing reality. Conventions are not 
understood as customs or traditions, but as institutional logic on how 
to interpret, evaluate and valuate (or valorize) persons, objects and 
events. As the convention theorist Alain Desrosiéres (2008: 10) stated: 
“to quantify is to implement a convention and then to measure”. It is 
these measurement conventions which bring in the link between facts 
and values. EC/SC has argued that convention-based measurements 
can be evaluated by studying how they enable measurements, which 
enable collective action aiming for a common good. Measurement 
conventions are not only agreements about measurement procedures, 
but as Figure 2.1 shows, they are also agreements about the ontologies 
of what is to be measured, i.e., entities to be classified or to be quanti- 
fied (Centemeri, 2012). 

For EC/SC, measurement conventions are embedded in statistical 
chains, which are built by situations, in which different actors are in- 
volved in the production of data (Desrosiéres, 2000, 2009, 2011). Sta- 
tistical chains (or statistical production chains) are characterized by a 
division of labor and can be burdened by differently applied conven- 
tions, which are mobilized by different actors. The notion of algorithm 
is different to the notion of statistical chains at first glance, because al- 
gorithms are built up by sets of calculative steps to proceed a designed 
task. Statistical chains can also be conceived of a series of steps, but 
in many cases, the whole chain cannot be planned and governed by 
one rational or one actor only and the chain is distributed (Diaz-Bone, 
2016, 2017). But, as Dourish (2016) has argued, the elements or calcu- 
lative steps of algorithms can be distributed too. Kitchin (2017) has 
pointed to the fact that algorithms are linked to other algorithms and 
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Figure 2.1 Measurement conventions. 
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build up “algorithm systems”. All in all, the consequence is that data 
and algorithm-based data analysis is distributed in networks of com- 
puters, organizations and persons — as in many big data analyses. This 
way, calculative changes, incoherencies and problems of data quality 
and adequacies can emerge in algorithmic procedures too. 


The Plurality of Data Worlds and Data Regulation 


Another pragmatist core notion is that of “worlds”. In EC/SC, this 
notion is applied to the plurality of “data worlds” (Diaz-Bone et al., 
2020; Diaz-Bone and Horvath, 2021). In these data worlds different 
orders of worth and quality conventions (Boltanski and Thévenot, 
2006) are combined and serve as rationalities for the evaluation of 
standards and data quality, but also for the link between data pro- 
duction, distribution and analysis on one side and specific common 
goods on the other side. Different data worlds can be identified as 
ideal types, which are characterized by different combinations of 
these orders of worth and quality conventions. In Western countries, 
these data worlds coexist and can be conceived of as different institu- 
tional rationalities, which are the blueprints for real organizational 
settings. The data worlds of official statistics, of academic science, of 
the civic data worlds are the most visible ones. Another data world 
is the big data world, whose algorithms and practices of data analy- 
sis are mainly invisible. This set of mentioned data worlds does not 
claim to be a complete one, but it has been useful so far to serve for 
contributions to a sociology of social research.4 And these worlds are 
in transition for different reasons. One reason is the tension between 
them, because in many situations these actors from some of these data 
worlds criticize practices, standards and effects of other data worlds. 
Another reason is that these data worlds cannot rely equally on data 
generating infrastructures, such as Internet platforms, and are influ- 
enced in different ways by new technologies, such as artificial intel- 
ligence and intense development and usage of algorithms. Some of 
these data worlds can be briefly sketched. 

The oldest data world is the academic science data world, which 
can be clearly identified with universities and research institutes. 
Here, the industrial convention and the convention of inspiration 
are most influential. The academic data world aims to generate new 
methods and knowledge to improve technologies (most visibly with 
engineering and computer sciences) as well as to advance mankind. 
This data world influenced the following ones, but still sticks to its 
claim to be impartial and not to legitimize itself with economic or 
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public engagements. The data world of official statistics emerged in 
the course of the last one and a half centuries. It co-emerged with 
industrialization and the development of statist bureaucracy. Official 
statistics is in most countries based on specific law, which equipped 
the institutions of official statistics with a high degree of legitimacy 
and power. Official statistics data worlds can rely on public adminis- 
tration as data generating and data sorting infrastructure. It delivers 
data reports and publishes indicators with a long-term orientation 
and with a high degree of visibility. 

The official statistics world is mainly structured by the industrial 
and the civic conventions and has long aimed to support but also to le- 
gitimate state policies. In its beginnings, official statistics was oriented 
mainly toward state administrations but it reoriented more and more 
toward national and international publics. One reason for this was the 
critique of the civic society, which claimed for a more public service 
orientation of official statistics. The civic data world can be conceived 
as emerging from social movements, non-governmental organizations 
(NGOs) and other civic actors and civic agencies, which collect and 
generate data to report on social issues and to build up empirical ev- 
idence for their claims. The civic data world is mainly influenced by 
combinations of the civic convention, the industrial convention, the 
domestic convention and the network convention. This world aims to 
engage for civilian’s participation and its identity is based essentially 
on forging a countervailing power against private and administrative 
forms of governance as well as to build up agency and critical capaci- 
ties on data and data transparency. A more recent form of engagement 
is data activism, wherein scientists and civic actors cooperate to detect 
unfair and illegitimate forms of data-based governance, identify un- 
intended and irrational consequences of software and algorithms and 
work on the application of algorithms to support civilian engagements 
(Milan and van der Velden, 2016; Didier, 2018). Also, organizations 
in this civic data world try to critically scrutinize algorithms applied 
to civilians and public issues and claim for transparency and public 
deliberation of algorithms.® Initiatives such as “open source”, “open 
data”, “open science” (Kitchin, 2014; Baack, 2015) or “citizen science” 
(Franzosi and Sauermann, 2014; Kitchin, 2014) can be regarded as 
part of this data world as well as engagements to build up more flexi- 
ble and more adequate data infrastructures to generate data, which is 
useful for public agency (Lane, 2020a, 2020b). 

The big data world is in some important respects different from 
the official statistics data world and the civic data world, because 
it exploits the privatized access and control of big data generating 
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infrastructures as Internet platforms and data markets. Also, data 
analysis is run by intensely applying advanced techniques such as de- 
veloping new algorithms and applying machine learning techniques 
and artificial intelligence (Mayer-Schénberger and Cukier, 2013). An- 
other characteristic is that its techniques, algorithms and data analyt- 
ics are not only handled as private companies’ assets, but these devices 
are also opaque and invisible to the public. Although companies from 
the big data world (most notably Google) provide free services to the 
public, this data world is criticized by the civic data world for the opac- 
ity of its algorithms and the privatization of its huge amounts of data. 
It is an open (and contested) question whether the big data world does 
aim for a common good. Evidently, the companies aim for profit in 
this data world. Even if all data worlds have to apply algorithms (¢.g., 
in statistical data analysis software), it is the big data world which is 
criticized for aiming to analyze and to influence individuals’ behavior 
(Zuboff, 2019). 

Nowadays, one can find new alliances between these sketched 
worlds and also combinations. For example, more and more state de- 
partments apply advanced algorithms and big data analytics to de- 
tect criminal behavior or to survey their citizens. But it is also state 
departments which align with civic agencies and NGOs to enhance 
public capacities to understand algorithms, their effects of public life 
and to advance public concerns.’ Actors from the civic data world, the 
academic science data world and the official big data world try to form 
coalitions to implement new data infrastructures as alternatives to the 
privately owned Internet platforms (Kitchin, 2014; Owen-Smith, 2018; 
Lane, 2020a, 2020b). 


Analyzing Algorithmic Regulations: Critique in 
Situations of Uncertainty 


The totality of rules and regulations to which people are exposed in 
digital contexts can be explained by the dynamics of conventions, rights 
and normative systems. As has been shown, these norms are partly 
visible, but partly hidden in technical contexts. The entanglement of 
different areas of society in which norms are set and in which shifts in 
norm-setting occur has led to a complex structure of regulators that 
is quite difficult to grasp at times. Formal law is set by states, digital 
platforms are regulated in their terms and conditions and normative 
ideas are inscribed in codes and algorithms. Especially in recent years, 
the debate around these issues has led to a number of regulatory pro- 
jects, transparency efforts and clarification of responsibilities. 
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To understand how actors try to cope with black boxes, different 
normative approaches and new technologies, it is critical to distin- 
guish between different coordinative goals and ideas of the public 
good. These are factors that come to play amongst different actors 
in — and coalitions between — data worlds during efforts of reshaping 
data structures and establishing transparency and accountabilities in 
a world that seems to be dominated by logics of commercial big data 
companies and their opaque deployment of algorithms. 

A key to understanding the critical capacities (Boltanski and 
Thévenot, 1999) of such actors and therefore the encounter of norma- 
tive and justificatory differences is the situational visibility of algorith- 
mic decision-making (ADM) processes, be it in situations of content 
moderation on platforms or algorithmic decisions in public adminis- 
trations. Whenever an actor is noticeably affected by such a decision, 
situations of uncertainty and critique may be observable. 

Taking the examples of content moderation and other surfacing 
ADM processes as an observable and therefore criticizable® part of 
algorithmic regulation, the described logics of data worlds and the 
EC/SC approach allow for casting light on the complex dynamics of 
algorithmic regulation and help reveal functionalities, necessities and 
justifications amongst the involved actors. 

The emergence and the power of the big data world and the eco- 
nomically driven large-scale usage of data in commercial contexts 
made private companies establish ADM processes as a conditio sine 
qua non.? Especially digital content-platforms brought the use of big 
data to perfection. Self-conceptualized as neutral places where peo- 
ple meet and interact, large amounts of individual and general data 
were collected and analyzed to improve both user experiences and ad- 
vertisement. Here, the opacity of algorithms is very high, and users 
usually won’t be aware of sorting algorithms, the categorizations and 
quantifications that strongly shape their experience online. 

Established and approved in micro targeting, advertising services 
and categorization, ADM also appeared to be the perfect choice when 
interventions on content and users were no longer viable (Gillespie, 
2020). Broadly labeled as content moderation, all interventions toward 
media, text or observable interactions of users have in common that 
they constitute direct and at least partially visible interventions on us- 
ers and their actions online (Gillespie, 2020; Gorwa et al., 2020). 

Even though there have always been intrinsic motivations for 
companies not to have certain content!” on their platforms, when it 
later came to questions of copyright (Perel and Elkin-Koren, 2016: 
484) or speech restrictions (Heldt, 2019: 3—4), the urge to moderate 
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content on a larger scale came — generally speaking — mostly from 
external pressure by governments and the civil society. Platforms 
found themselves in a tension between different ideas of public good 
that were brought to them externally. Together with their very own 
ideas of restricting certain content, a complex environment of mod- 
erating and regulating regimes was established on most of the large 
platforms for user-generated content such as Facebook, Twitter, 
YouTube or Instagram (Gorwa et al., 2020: 6) but also in search en- 
gines and intermediaries like payment service providers (Tusikov, 
2017: 22-25). 


That platforms operate their moderation practices under a com- 
plex web of nebulous rules and procedural opacity renders situ- 
ations [...] even more challenging, with governments and others 
clamoring for tighter controls on some material, and other mem- 
bers of civil society demanding greater freedoms for online ex- 
pression [...]. 

(Roberts, 2018) 


The problematic notions of online content moderation are just one of 
several examples, where ADM processes (with their hidden values and 
normativities) are directly interwoven with concrete ideas of norms 
and norm enforcement. When it comes to governmental decisions, 
as in public administration or organizational procedures like human 
resource management, the use of big data approaches offers easy, 
scalable and reliable solutions but remains linked to the problems of 
opaque norms, values and measurements within automations. 

Such situations where actors are notably affected by algorithmic 
decisions could generally be grasped as situations of uncertainty for 
different actors. Herein, a plurality of requirements and logics of reg- 
ulation, measurement conventions and automation culminates in sit- 
uations where algorithms may still operate opaquely but not unseen 
in their outcomes. On an individual level, human attention might first 
and foremost be paid by the moderated user, whose tweet, YouTube 
video or Facebook post was blocked, deleted or labeled as problematic 
(Myers West, 2018), by a citizen who was denied a public service (AII- 
hutter et al., 2020) or an employee who was dismissed (Soper, 2021). On 
a larger scale, the outcomes of content moderation or other problem- 
atic automated decisions are situationally structured as well (Barthe 
et al., 2013: IV) and may come to the attention of platform operators, 
government officials or NGOs. It is important to mention that the 
neopragmatist notion of situation does not only cover configurations 
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occurring in simultaneous moments and places, but also distributed, 
stretched and interrupted constellations in which actors coordinate 
their actions (Diaz-Bone, 2011: 49). 

EC/SC has mobilized situations of uncertainty as observation 
points, where actors are confronted with situations that require a con- 
scious decision, which must be justified with a reference to a specific 
kind of common good usually linked to conventions. As such situa- 
tions of norm enforcement do not only affect actors of the big data 
world but also single users, the state and the civil society, other justi- 
fications than those from the mere big data logic, must be brought up. 
This causes an epistemological breaking point that could be utilized 
as a potential bottle opener for some of the opaque logics that are hid- 
den within algorithmic black boxes. As such situations however won't 
be able to explain the hidden norm-based quantifications and classifi- 
cations within ADM processes entirely, they offer a leverage point to 
force engaged actors to justify their actions. 

As big data has brought the usage of algorithmic regulation and 
automated decision-making to a standard in regulating, moderating 
and sorting large amounts of data, the promises of objective, fair and 
reliable data-based decisions convinced not only commercial actors 
but also governments. With their history in official statistics (Des- 
rosiéres, 2009, 2011) and an interest in both enforcement and efficiency 
(Engstrom et al., 2020: 22), governmental actors are principally open 
to the employment of similar approaches of decision-making, often di- 
rectly offered by private companies (Engstrom et al, 2020: 7). Mainly 
structured by the industrial convention and the civic convention, the 
market convention (as in the big data world) is less applicable in bu- 
reaucratic situations. Nevertheless, problems of training data quality, 
incoherencies and classificational errors persist. Such problems may 
be marginalized and less problematized in commercial contexts but 
are of greater importance and target of critique within legally defined 
boundaries of public administration. States are usually subjected to 
greater demands and duties for transparency and accountability than 
private firms. Also, these transparencies enable actors from the civil 
society to criticize and intervene in state-driven projects. 

Actors from the civic data world have not yet distinguished them- 
selves through the extensive use of ADM processes on their own, but 
their expertise and their common good-related goals such as freedom, 
nondiscrimination and equality are important when it comes to initi- 
ating criticizable situations on a large scale. NGOs collect single-case 
reports and workout reports, e.g., on structural discrimination, biases 
and in transparencies, both in private and public application areas. 
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Meanwhile, an important source of justification for such critique 
often comes from a data world that is at the same time involved in 
developing the technical foundations and, because of its profound 
knowledge, is also aware of their limitations and problems. Unlike the 
big data world, the academic science data world is strongly interested 
in epistemological knowledge and less in the (commercial) usability of 
knowledge. Trained in theories and philosophy of science, the implica- 
tion of falsifications and epistemology, academic data scientists are an 
important source of critique when it comes to tackling the very base of 
big data (Kitchin, 2014, 2017; Symons and Alvarado, 2016), promoting 
the idea of critical data studies in science (Iliadis and Russo, 2016) 
and education (Pangrazio and Selwyn, 2021) or even pointing out the 
linkage between norms and measurements (Desrosiéres, 2008, 2014; 
Diaz-Bone 2016, 2017, 2019; Diaz-Bone and Didier, 2016; Diaz-Bone 
and Horvath, 2021). Despite a greater access to and understanding of 
the structure of algorithms (Zweig et al., 2018; Krafft et al., 2021), this 
kind of academic critique also relies on situations of uncertainty, even 
though they are intentionally brought to light by researchers to chal- 
lenge platforms and governments: 


To hold software service and platform providers accountable, it is 
necessary to create trustworthy, quantified evidence of problem- 
atic algorithmic decisions, e.g., by large-scale black box analyses. 

(Krafft et al., 2021: 143) 


When it comes to algorithmic regulation, data worlds are entangled 
in a complex constellation of reciprocal critique and sometimes find 
themselves in processes of norm setting or duties to enforce norms 
— depending on their different capabilities to regulate. But how does 
critique in situations of uncertainty shape actual regulatory regimes 
and how do actors coordinate and justify their actions in this regard? 
The next section will bring this issue to life by examining selected em- 
pirical phenomena in which ADM processes are applied. 


New Regulators — New Perspectives on Regulatory Processes 


While successfully pursuing goals of data-driven business such as 
surveillance, ad targeting and general data collection, big data actors 
found themselves as regulators on their own and de facto enforcers 
of legal and other external norms. Using often opaque and therefore 
hardly criticizable automation, several areas of the digitalizing soci- 
ety are affected by new and not thoroughly overviewable regulatory 
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processes. There is more than just a duality of regulators — state and 
big data actors — who determine the regulatory environments that 
users, citizens, employees or customers are confronted with in digi- 
tal everyday life. It is the complex interplay between data worlds and 
those who act and react within them who justify, criticize, evaluate and 
develop not just technically but more importantly convention-based. 
Situations of uncertainty provide a leverage point to observe such ac- 
tion, coordination and critique. 

The proposed scheme to analyze new regulatory areas and processes 
by mobilizing the critical capacities of actors within data worlds and 
the conventions they rely on promises worthy insights into several ar- 
eas of algorithmic regulation. The following five are a cursory account 
of some of them, with suggestions which situations of uncertainty 
do or may come up and how critique is made possible and justified 
by ideas of common goods. This should provide an overview of how 
broad and widespread both the usage of ADM processes and the po- 
tential of its convention-based analysis are. 


1 Big-data platforms and the enforcement of copyright and speech 
regulations are the most notable examples when it comes to the 
large-scale enforcement of legal norms by online service provid- 
ers. Uncertainty may occur for users who post or upload disput- 
ably (illegal content. At the same time, there is uncertainty (e.g., 
for rights holders) on whether they get royalties for copyrighted 
materials (Leistner and Metzger, 2017) or for those who are ad- 
dressed by disputably offensive speech (Heldt, 2019). As the big 
data world tended to justify their automated moderation decisions 
on copyright and hate speech (Gorwa et al., 2020: 6) from an in- 
dustrial convention, critique is mainly fueled by civic data world 
actors, e.g., NGOs, but also the scientific data world come from a 
civic and domestic convention. As algorithms are currently said to 
come to their limits in borderline cases (Elkin-Koren, 2017), such 
as sampling, quotes or satirist or pastiche content, state regula- 
tors, e.g., in Europe, started to force platforms to install ex post 
of human-based out-of-court resolutions (European Commission, 
2020; Cauffman and Goanta, 2021: 12-13; Quintais and Schwemer, 
2021: 16) and ex ante possibilities to pre-flag, e.g., fair-use content 
(Metzger and Senftleben, 2020: 128). This could be grasped as an 
endeavor to enable transparency for outcomes of algorithmic de- 
cisions and to enable and inform users to act in such uncertain sit- 
uations. At the same time, in a similar ex post fashion, Facebook 
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installed its own widely discussed (Douek, 2019; Klonick, 2019; 
Golia, 2021) oversight board to criticize, evaluate and oversee its 
own actions in an alleged independent manner. Such an action 
acknowledges the need for insights, critique and control, but does, 
as critics put it, not adequately answer questions on independent 
control. 

The vast amounts of regulatory power do also come with their 
own uncertainties as big data platforms and enforcement regard- 
ing child sexual abuse imagery and terroristic content illustrate. 
As there is no dispute on the common good of not wanting to 
have such content available, questions of the borders’ enforce- 
ment and surveillance come up and must be justified. After suc- 
cessfully developing and implementing automated recognition 
systems for abuse imagery such as PhotoDNA (Farid, 2018) or 
terrorist content such as the Shared Industry Hash Database 
(Gorwa et al., 2020: 2), there may be uncertainties and need for 
justifications for online service providers where to technically 
stop surveilling and how to handle different qualities of nonbind- 
ing requests from governmental officials, binding court decisions 
(Fisherman, 2019: 90) and soft requests from the civic society 
(Bursztein et al., 2019: 1). Such delicate topics are also a source of 
uncertainty for civic actors as NGOs who criticize surveillance 
but agree with the goal of removing abuse imagery and terrorist 
material (Perez, 2014). 

While the cases mentioned above have their restrictions legally 
defined, there are socio-normative conflicts within the big data 
world, e.g., ethically driven disputes between porn platforms and 
payment services where platforms regulate in different but highly 
impactful ways that are partially dispatched from legal questions. 
The repeatedly surfacing conflict (Tusikov, 2021: 73-74) between 
payment service providers such as PayPal, Visa or Mastercard and 
porn platforms, most notably Pornhub, revolving around the ac- 
cessibility of payment options for users and actors and vice versa 
the availability of user-uploaded, unverified content. Incidents 
such as the suspension of payments or the instant deletion of over 
ten million nonofficial videos by Pornhub in 2020 (Kastrenakes, 
2020) showcase the enormous regulatory power such platforms 
have, especially in grey areas of society (Tusikov, 2021: 75-76). This 
conflict is coined by reciprocal uncertainty whether it is ethically 
justifiable to offer payments for adult content or whether pornog- 
raphy business models for both actors and platforms could prevail. 
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4 


As examples 1-3 illustrate the potential of big data actors as im- 
portant regulators and enforcing actors, another impactful and 
critical application field of ADM is public administration and the 
usage of big data technologies. As mentioned, public adminis- 
trations that are connected to the world of official statistics and 
therefore classifications of citizens are structurally open for the 
application of big data-driven sorting and decision-supporting 
algorithms that either directly stem from private big data compa- 
nies (who often exclusively have the data required to offer certain 
services) or are developed under comparable premises and con- 
ventions. While administrative acts underlie more explicit public 
accountabilities and formal law, ADM systems are more and more 
in use by state agencies (Engstrom et al., 2020). Critical questions 
in regard to accountability and explainability are crucial in dem- 
ocratically controlled administrative processes: 


When public officials deny benefits or make decisions affect- 
ing the public’s rights, the law generally requires them to ex- 
plain why. 

(Engstrom et al., 2020: 7) 


The hurdles for an implementation of such technologies appear 
higher, which is mainly due to the strong civic convention and 
the common good of a nondiscriminatory and transparent de- 
mocracy. In such cases, formal law simultaneously functions as 
a coordinative framework and a guardrail for those who imple- 
ment. That critique from external actors from the academic or the 
civic data world remains crucial, is shown by several cases, e.g., 
where authorities have used prediction algorithms in Austrian 
job centers (Allhutter et al., 2020), language recognition pro- 
grams in German asylum processes (Keiner, 2020) or privately 
run person-based technologies in predictive policing in Canada 
(Robertson et al., 2020: 47-50) that sparked public critique and 
controversy and forced actors to justify their actions. Situations 
of uncertainty come up amongst those who are affected by such 
decisions, e.g., when assessments are wrong or predictions are 
discriminatory or biased. In the course of public outcry, uncer- 
tainty emerges for public administrators who have to justify the 
use of such technologies without (sometimes) being fully capa- 
ble of seeing through the opaque interior of both databases and 
algorithms. 

The last of the five application areas is at the very intersection 
between commercial, formally legal and civic logics and an in- 
teresting outlook on further challenges of ADM. Legal tech is an 
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intersection between big data and legal norms. Legal tech describes 
the application of big data technologies on legal services. Even 
though the large-scale use of legal tech is still in its early stages, 
here different dynamics are observable than in the other areas men- 
tioned above. As legal tech is per definitionem strongly interwoven 
with legal norms, there is no notion of ex post interference of law 
with existing business models as in the mere big data world, but law 
itself becomes a target of applying such logics. Turned this way, 
law and big data ADM seem to be made for each other. Both ap- 
pear to rely on a logical, structural framework where step-by-step 
transformations from data into outputs are made (Pasquale, 2019: 
1). Since the beginnings of computer technologies, law has been 
subject to (usually academic or political) endeavors of cybernetics 
and digitization (Salami, 2017). With commercial approaches and 
ADM technologies, the digitization of only seemingly simple legal 
processes such as legal forms, contract generation or contesting 
speeding tickets (Pasquale, 2019: 12-18; Hähnchen et al., 2020: 631) 
started to become a business model. 

The big data world again promises efficient, precise and scalable 
automations that find fruitful soil amongst commercial branches 
of legal administration such as law firms and lawyers (Hartung 
et al., 2018). Academic scholars discussing the potential of legal 
automation use their insights in both legal and computational 
processes to confront big data promises with critique from civic 
and domestic conventions that point out not just misconceptions 
of the formal logics of legal syllogisms (Pasquale, 2019; Hahnchen 
et al., 2020) but general uncertainties for legal professionals, cli- 
ents and plaintiffs who may face typical imprecisions, opacities, 
simplifications or even biases that come with ADM (Engstrom 
and Gelbach, 2020: 1024). The common good of due process and 
the specially protected attorney-client relationship could hardly 
be upheld when big data scalability and automation are central 
coordinative goals (Pasquale, 2019: 60). 


With these five exemplary fields of application in mind, EC/SC’s 
perspective view on regulatory configurations and their situational 
settings casts a light on the intertwining of state and private regula- 
tion, as both of them are more and more enforced by ADM processes 
and therefore dominated by big data logics and their inherent conven- 
tions. While those processes come with contested yet often convincing 
promises of objectivity, efficiency and scalability, their opacities and 
lacking accountabilities become an emerging problem for otherwise 
well-established controlling mechanisms. 


40 Rainer Diaz-Bone and Simon Schrér 


Critique and reciprocal justifications, brought up by directly in- 
volved actors or actors capable of overseeing larger contexts, depend 
on information and clarification. The approach of utilizing situations 
of uncertainty to formulate critique and demand justifications helps to 
identify not only existing conflict points but shows strikingly how nec- 
essary it is to have constant accountabilities not just in a legal but also 
a social manner. As unsolved problems with biases, discrimination, 
erroneous decisions and the blending of commercial and common 
good goals show, the important mechanisms of critique and justifica- 
tion are weakened when observability is either impossible or optional 
and only occasionally admitted. 


Notes 


1 Technically seen, an algorithm is a finite set of calculative steps, which 
are organized in an ordered sequence (that allows a selection of steps and/ 
or loops) and which transforms a data input in an output. An algorithm 
is realized as a code sequence in a programming language, 1.e., software 
(Louridas, 2020: 19/23/26). 

2 The term “algorithmic governance” was introduced by Miiller-Birn et al. 
(2013); see, for a sketch of this notion and its genealogy, Katzenbach and 
Ulbricht (2019). 

3 As O’Neil therefore has stated, algorithms can be conceived as “an opin- 
ion formalized in code” (O’Neil, 2016: 49). 

4 See for more details Diaz-Bone et al. (2020) and Diaz-Bone and Horvath 
(2021). 

5 See, e.g., the movement of “Statactivisme” in France (Bruno et al., 2014; 

Didier, 2018), “MyData” in Finland (Lehtiniemi and Ruckenstein, 2019) 

or “DataKind” in the USA. 

One example is “Algorithm Watch”, see https://algorithmwatch.org/en/ 

One example in Germany is Civic Coding, see https://www.civic-coding de/ 

In the sense of critics by enlightened actors as Boltanski (2011) describes it. 

Even mass-scale human decisions as in click working content moderation 

is deeply embedded in preceding algorithmic or AI sorting and deciding 

what is to be displayed to the human moderators. For Facebook, see, e.g., 

Vincent (2020). 

10 Such as pornography, violent content or spam. 


Oana 
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